👀 Problem

DePINs are currently ranked by total market cap:

• https://depinhub.io/rankings/categories

• https://coinmarketcap.com/view/depin/

• https://www.coingecko.com/en/categories/depin

• https://coinranking.com/coins/depin

• https://depin.ninja/

This method ranks networks by value based on speculation, and not based on demand. In other terms, it ranks networks by investor attraction, and not by usefulness. Can Storage DePINs actually be used?

💭 Proposal

Storage DePIN Leaderboard

Checker Networks wants to turn around the DePIN market, getting from being interesting to being useful. One way of getting there is by building a Leaderboard like the ones listed above, but based on usefulness.

Since Checker initially focuses on Storage DePINs, this document proposes a Storage DePIN leaderboard.

Value flow

A leaderboard will create competition between DePINs, incentivising networks to be on top of the list.

A leaderboard also makes DePINs more accessible, as it helps consumers select the right DePIN network for their use case.

In turn, DePINs will want to buy detailed analytics from the leaderboard data source, in order to ensure the highest possible rank.

In order to ensure coverage of their whole DePIN network, DePINs will need to fund the Checker Network, in order to pay operators

graph TD
  Leaderboard -- Incentivize usefulness --> DePINs;
  Leaderboard -- Drive demand --> DePINs;
  DePINs -- Buy analytics, monitoring, alerts --> CheckerNetwork;
  DePINs -- Fund operators --> CheckerNetwork;
	

TODO: @Patrick Woodhead please help complete this section

How to rank Storage DePINs

One aspect of building a leaderboard is the ability to compare competitors. Checker is currently only measuring one DePIN Storage network, and that is Filecoin.

Current metrics

For Filecoin, the main metric is:

• Retrievability: Whether a piece of data can be retrieved

Two auxiliary metrics are being exposed:

• Retrieval Result Codes: A breakdown of retrieval results, helping in debugging Retrievability

• Time to First Byte (TTFB): A performance indicator for retrievability

See https://dashboard.filspark.com/

For ranking Storage DePINs, we need to define a set of measures that are independent of Filecoin and can be applied to allStorage DePINs.

Metric criteria

A leaderboard metric should be

• fair: It shouldn’t favor one Storage DePIN over another

• simple: For a Storage DePIN customer, it should be easy to understand

• relevant: For a Storage DePIN customer, it should be relevant in picking a solution

• universal: It should be straight forward to apply the metric to any Storage DePIN

• resilient/distributed: Even if a DePIN adds centralized infrastructure to improve usability, Checker should test the system in its most distributed form possible. This helps incentivize resilient and robust solutions.

Metric selection

Since Storage DePINs are still in an early stage, the question of whether a piece of data can be retrieved at all is a central question. Therefore, the first goal is to Retrievability in a way that can be applied to all Storage DePINs.

The usefulness of the Leaderboard itself should be measurable through this metric alone.

We can further deepen the offering by adding deeper metrics, like pricing / TTFB / retrieval speed. This is left open for a future work stream.

🛫 Plan

1. Define Retrievability metric, adhering to the metric criteria laid out above
   1. Bonus points if it’s similar enough to the Spark definition that no change to the Spark subnet is necessary

2. Collect Retrievability metric from all Storage DePINs currently measured by Checker. At the time of writing:
   1. Filecoin

3. Build out pipelines so that Checker is measuring Retrievability of 3 or more Storage DePINs. There need to be enough competitors for the ranking to be relevant.

4. Create a public leaderboard website

   

5. Gather early feedback from selected competitors

6. Announce leaderboard

Questions

• Should market cap be part of the Checker Leaderboard?
  • J: A DePIN network is stabilized, incentivized and regulated through its token. Is a network with too low market cap risky to use?

• Should network size be part of the Checker Leaderboard?
  • J: ❗ A network with 100% retrievability but only 1 node is barely useful → Yes